Contractions: Breaking the Tokenization-Tagging Circularity

Ambiguous strings are strings of non-whitespace characters, typically coinciding with orthographic contractions of word forms, that depending on the specific occurrence, are to be considered as consisting of one or more than one token. This sort of strings is shown to raise the problem of undesired circularity between tokenization and tagging. This paper presents a strategy to resolve ambiguous strings and dissolve such circularity. 1 Tokenizing-Tagging Circularity As a starting point, let us take orthographic contractions in the most widely studied natural language. In spite of the fact that in English, these items do not exhibit a large diversity, the few existing cases are illustrative examples of the need to identify different tokens out of a single sequence characters not including whitespaces. For instance, in the Penn Treebank, the sequences I’m and won’t are tokenized as |I|’m| and |wo|n’t|, respectively [2]. If for the sake of facilitating subsequent uniform syntactic processing of every verb form, orthographic normalization is sought, then reconstruction of correct orthographic form of the identified tokens is also an issue here and the strings I’m and won’t should be tokenized as |I|am| and |will|not|, respectively. In Portuguese orthography, there are several instances of orthographic contractions. Most relevant cases concern the contraction of a Preposition with the subsequent word. Some Prepositions contract with Articles and also Personal and Demonstrative Pronouns: pelo (por o), nele (em ele), desse (de esse). Besides Prepositions, also some Clitics either in proclisis or not, may be contracted with other clitics: lho (lhe o). In this connection, the crucial point to note is that some of these strings are ambiguous between a contracted and a non-contracted form (see list of ambiguous strings in Table 1). For instance, the string pelo is ambiguous between the first person singular of “Presente Indicativo” of Verb pelar and the contraction of the Preposition por and the Definite Article o (for the full list of ambiguities, see [1]). 168 A.H. Branco and J.R. Silva These items introduce circularity in the standard scheme of tokenizationfollowed-bytagging: They should have been previously tagged for them to be correctly tokenized – but for them to be tagged, they should have already been tokenized. For instance, to decide whether mas should be tokenized as |mas| or as |me|as| in a specific occurrence, it is necessary to know whether in that specific occurrence mas is a Conjunction or the contraction of two Clitics, respectively: This presupposes that the tagging process should have taken place, but for this process to have taken place, tokenization was expected to have been applied, and the circle is closed. The low level of the task of tokenization with regards to the spectrum of the NLP levels should not bias one to not recognize the importance of this problem. From a performance point of view of any NLP task, this is far from being a minor or merely curious issue. On the one hand, these ambiguous strings include potential tokens of functional categories, which are known to be of the utmost importance to constraint the syntactic environment where they occur and to help guiding the subsequent parsing process. On the other hand, these items have a non-negligible frequency, accounting for as much as around 2.1% of a large enough corpus, as the one we used in the experiment reported in the next section.1 A careless, low precision approach to tackle this issue, at such an earlier stage of processing, is very likely to imply a severe and unrecoverable damage in the overall performance of subsequent tasks, and in particular, in the immediately subsequent task of tagging. The importance of an accurate handling of the ambiguous strings can be shown in detail by taking into account their relative frequencies and the relative frequency of each of the two options for their tokenization in a corpus:2 Table 1. Distribution of ambiguous strings Strings Freq. One token Two tokens consigo 17 8 9 desse 33 6 27 desses 14 0 14